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ABSTRACT 

CNU Online is the computer-managed instructional 
delivery system of Christopher Newport University. Fully functioning 
to deliver wholly online courses since fall 1994, CNU Online has been 
using an officially approved assessment plan cince January 1995. The 
plan is presented as a useful example for others developing online 
instructional delivery. The goal of the plan is to provide evidence 
of effectiveness in meeting the goals of CNU Online by using accepted 
standards of assessment that are consistent with state and regiona 
euidelines. CNU Online must meet the defined state and regional 
standards for all on-campus programs with regard to assessment goa s 
and quality. The levels of assessment include: (1) online program an 

institutional efficiency; (2) learning outcomes; (3) teaching 
effectiveness; and (4) student satisfaction with course delivery. 
Goals are defined for each of these areas, and methods for evaluation 
are outlined. These include standard surveys and more innovative 
approaches to evaluation. Some collateral assessment activities are 
suggested. Appendixes present an instructional evaluation survey, and 
a rating scale for peer group-based juried review of aspects of 
instructors' message logs. (SLD) 
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ASSESSMENT PLAN FOR GNU ONLINE 



PREFACE 



Online is the computer-managed instructional delivery 
system of Christopher Newport University. Fully functioning to 
deliver wholly online courses since fall, 1994, CNU Online has been 
utilizing the following officially-approved "Assessment Plan for 
CNU Online" since January 14, 1995. This assessment plan is hereby 
offered for broad dissemination as a helpful contribution to the 
burgeoning field of online instructional delivery. 



ASSESSMENT FL20I FOR GNU ONLINE 



Introduction and Rationale 

The goal of this assessment plan is to provide evidence of 
effectiveness in meeting the goals of GNU Online by employing 
accepted and approved standards of assessment which are consistent 
with state (SCHEV) and regional (SACS) guidelines. Since the 
program being evaluated is new and highly innovative, it is 
expected that the assessment used will also require resourcefulness 
and innovation. 

The plan was also designed to deliver results. What follows 
is the rationale for the innovative approach which was adopted, 
with the interests of both sound methodology and effective 
implementation of assessment results fully in view. 

Assessment processes provide the linkages that connect diverse 
parts and strata of institutions and account for aggregate 
consequences of individual behavior. Institutional behavior is 
understood as the product of individual actions. From a policy 
perspective three stages of analysis focus on the operational level 
(e.g., what courses will be taught by whom, using what pedagogy, at 
what time and place) ; the collective choice (e.g. , what rules will 
be made affecting retention, promotion, and tenure of personnel to 
administer instruction in what disciplines) ; and the constitutional 
level (e.g., the rules for rule making that govern operational and 
collective choices) . 

The decisions of a given level set the institutional rules for 
the next lower level. At the constitutional level the mission 
statement and the University Handbook (including rules for the 
performance appraisal system) set the basic boundaries for making 
collective choices about personnel qualifications and behaviors. 
The decisions made at the collective choice level determine the 
information and resources which govern decisions at the level of 
academic departments. 

The constitutional rules identify the elements of what 
constitutes student satisfaction, the measures of effective 
teaching, the designated learning outcomes deemed to be a public 
good, and aggregate indicators of program efficiency which serve as 
bases of comparison with outputs of peer institutions. 

The application of these constitutional rules provides the 
framework in which specified information and data in dossiers 
submitted to peer groups, deans, provost, and president are used to 
make collective decisions about individual and institutional 
effectiveness in achieving designated learning outcomes. Feedback 
from these collective processes is given to academic departments, 
committees, and academic support units to guide policy making and 
opGrational decisions at the level of consumer servxces. The 
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Undergraduate Academic Program Review Committee, to achieve systems 
integration, could focus more on the interface and service needs of 
relevant departments. The result of implementing these processes 
would be to use assessment data to promote interlevel linkages and 
to produce in the aggregate an integrated and coordinated service 
product to the consumer-publics. 

Peer group and administrative review are little more than 
words on paper until they affect the behavior of course instructors 
and student target groups. 

The particular usefulness of this model is that it delineates 
the components of a process made up of attributes of the individual 
actors and the "decision situation." Individuals can find 
themselves in institutional arrangements and in a relevant 
community of other persons who have a stake in the decisional 
outcome. Individuals make choices based on their valuations of 
events and issues in the decisional situation confronting the 
institution. The assessment process and its findings provide 
incentives and constraints and result in the effective use of 
resources. 

Both the timing of the assessment process and the character of 
its provisions will satisfy methodological rigor and scholarly 
integrity. In order to meet deadlines and operate within budget in 
the short run, there may need to be a phased implementation of the 
full assessment package described below. The implementation 
schedule ultimately must be consistent with the minimum reporting 
guidelines set forth in the original terms of the GNU Online budget 
initiative document. 

Assessment plans and procedures described in this document are 
subject to review and revision in the light of subsequent 
experience. A review of the plan will be undertaken annually. The 
first revised document detailing assessment procedures will be 
circulated no later than December 15, 1995. 

STATE AND REGIONAL GUIDELINES REGARDING ASSESSMENT 

State and regional guidelines are relevant to assessment of 
student learning and performance and to the broader evaluation of 
programs. At the present time GNU Online is regarded as part of 
the on-campus academic program since its design and implementation 
are locally controlled. As such, the guidelines applying to the 
on-campus programs also apply to GNU Online. 

Three principles are particularly germane to assessment as 
defined by state and regional guidelines. First, the institution 
must demonstrate how it will achieve its goals, particularly 
student learning goals, and maintain a high standard of quality in 
doing so. Accordingly, goals must be stated which are specific and 
assessable. Second, the assessment should provide assurance that 



standards of quality are successfully maintained at an appropriate 
level regardless of the medium of the course (i.e., electronic or 
other) or the methods of instruction adopted. Third, the 
responsibility for the conduct of assessment should be 
appropriately delegated and shared. 

Of the three principles, the first principle is well 
understood and requires no further comment. The third is covered 
by the designation of responsibility found in this document. Only 
the second requires further comment. 

SCHEV's concern as regards the second principle is clear and 
understandable. It is a concern that students have a reasonable 
assurance that the course offerings they believe they are taking, 
based on published descriptions, are accurate regardless of where 
or under what format the course is offered. In particular, the 
objectives of the course in terms of student outcomes will have 
some similarities (not total but clearly some similarities) , for 
courses having the same course name and description, whether the 
format is online or in the classroom. SCHEV's concern may have 
policy implications in the future concerning the encouragement and 
funding of programs to deliver educational services in the most 
cost-effective and efficient manner. Relevant data (primarily 
student— outcomes related but also student satisfaction measures) 
are needed to make such recommendations. Further, this type of 
information can be useful on the institutional level as well; i.e. 
evidence showing parity or disparity in performance and 
satisfaction may be used either to reinforce what we are doing or 
to modify teaching and learning to produce even more favorable 
results. 

Implementation of the SCHEV guidelines should focus on course- 
specific comparisons. The guidelines make reference to full 
programs and to courses which are part of a degree program. While 
both of these situations exist, for the present at least it is not 
the comparisons of programs which should be of paramount concern 
pursuant to these guidelines, but the course-by-course comparisons. 

Since assessment data resulting from the implementation of 
this plan will have both internal audiences and a primary external 
audience (i.e. SCHEV) , it is important to distinguish the audiences 
for purposes of dissemination of findings. Not all assessment data 
which are useful internally need to be included in reports to 
SCHEV. The decision regarding what is relevant to report to SCHEV 
will be based on the reporting requirements associated with the 
original budget initiative document and SCHEV' s legitimate concerns 
as outlined above. 
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BACKGROUND DATA 



Preliminary assessment should include developing a 
statistical description of students who choose to enroll in on-line 
courses (i.e. age, gender, ethnic, prior enrollment, GPA, 
classification, other background) . This type of information is 
essential to any attempt to account for background characteristics 
as a factor in student learning which is independent of program or 
course effects. Since the student characteristics associated with 
enrollment in on-line courses may be different from other students' 
characteristic profiles, these differences could help to account 
for apparent program and course-related effects or their apparent 
lack of effect. 



LEVELS OF ASSESSMENT IN INSTITUTIONAL CONTEXT 



This plan views the CNU Online program as integrally related 
to the CNU institutional context. There are distinct advantages in 
integrating assessment practices wherever possible into ongoing 
institutionalized procedures. One benefit is that assessment can 
then be carried out less intrusively and with generally greater 
acceptance and effectiveness. Another advantage is that the 
assessment data are generally more likely to inform and have the 
•influence on the institution that they warrant when they are 
integral to its established procedures rather than external to 
them. Thus, valuation in this plan takes place within the 
institutional context deriving information from four distinct 
levels and in turn potentially informing the institution in a 
number of ways. These levels include (from broadest to narrowest): 



Level 1 : 
Level 2 : 
Level 3 : 
Level 4 : 



Online program/ institutional efficiency 
Learning Outcomes 
Teaching effectiveness 

Student satisfaction with course delivery. 



For our purposes there are four levels of goals corresponding 
to the levels of evaluation. The following presentation includes 
the relevant goals followed by the assessment methods designed to 
measure their fulfillment. It should be stated at the outset that 
the anticipated use of information, by levels, will be as follows: 
levels 1 and 2 will be for uses both external and internal to the 
institution, and levels 3 and 4 will be reserved for internal use 
only. 
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I. ONLINE PROGRAM/ INSTITUTIONAL EFFICIENCY 
Relevant Goal Statements and Their Measurement 



The following goal statements apply to the entire GNU Online 
program. The main focus of the goals is upon the efficiency of 
service delivery. The methods of measurement of these elements are 
preliminarily defined, or an appropriate schedule for such 
development, appear under each relevant statement. The Office of 
Institutional Effectiveness will take the lead in this area. 
Appropriate parameters, variable definitions, and methods of 
measurement will be defined by April 15, 1995. 

1. Degree productivity will be enhanced and student retention 
will be increased. 

Degree productivity will be measured in terms of the number of 
graduates and the length of time to attain a degree. Student 
retention will also be examined over time both for students 
with the online degree intention and other students enrolled 
in online courses. However, the definition of variables, 
cohorts, and methods will be the subject of further 
delineation during spring, 1995. 

2. Accessibility for students with financial and mobility 
problems will be enhanced. 

Program accessibility will be measured in terms of convenience 
and necessity as identified by students. A complete catalog 
of constraints and limitations on access, and preliminary data 
on the prevalence of each, will be created during spring, 
1995. Further definition of variables, cohorts, and methods 
will be accomplished as needed during the same time period. 

3. Cost effectiveness and operational efficiency will be 
enhanced. 

The efficiency of the online program will be compared to the 
classroom program using a number of measures of cost. 
Included among the costs to be considered are: building and 
facilities, admissions, registration, financial aid, billing 
and payment services, veterans services, bookstore, career and 
counselling services, other student services, information 
management costs (lES, computer labs, information/ technology 
services) , FTE student/teacher ratio, and telecommunications 
costs. Cost effectiveness also will be measured by student 
opinion in terms of the importance of scheduling, travel, etc. 

However, the above is only a prelimary listing. Further 
definition cf \ariables and methods will be accomplished as 
needed during spring, 1995. 
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4. Student learning and academic advising will be enhanced by 
analysis of archival information from the digital environment. 

This area represents a serious objective for scholarly 
research which is likely to pay great dividends for practical 
application. The existence of extensive message logs of the 
CNU Online course experience provides a marvelous opportunity. 
However, this interest is long-term and will receive lower 
priority during the initial phases of program implementation. 

II. LEARNING OUTCOMES 

Relevant Goal Statements and Their Measurement 

Assessments at the second level attempt to provide dii*ect 
evidence of the fulfillment of the following relevant stuc.ent 
outcome goals. Operational measurement, and thus assessment, of 
goals 1-6 is also defined below and is further elaborated in the 
paper, "Teaching Online, Computer Managed Conferencing," by 
Williams, Teschner and Miller. 

Statements 1-6 below should be read beginning with the phrase 
"Students enrolled in CNU Online courses will..." 

1. Learn to read, interpret, and critique written texts without 
oral interpretation (vs. passive lecture and memorization) . 

Operational measurement will be performed in terms of two 
areas: 



a. instructor assessment of student growth in guided reading 
using unit learning objectives 

b. pre- and post-tests for each course module 



2. Learn to write clearly, analytically, and persuasively within 
a conceptual framework. 



This goal will be assessed in terms of templates which provide 
criteria for quality control of the work product. Examples of 
such templates are: 



a. 

b. 

c. 

d. 

e. 

f . 

g- 

h. 

i. 

j- 



purpose and scope of the essay 

paragraph organization as indicated by the scope 
statement 

summary of contents 

concluding assessment of topic significance 

pro-contra-reply 

comparative analysis 

cause and effect analysis 

alternative analysis 

concept application 

personal analogy 



k. structure-function analysis 

l. criteria identification and evaluation analysis 

m. case analysis 

In addition to the above, other appropriate templates may be 
developed to translate form into substance and provide 
criteria for evaluation of student work products. Online 
instructors are encouraged to use their disciplines to do so. 

3. Learn to work with peers creating problem-solving documents 
and form cohesive groups (teams and task forces) . 

This goal will be measured in terms of; 

a. appropriate complexity of division of labor among group 
members 

b. creativity of techniques for conflict resolution and 
application of negotiation skills 

c. (juality of analysis and content of work product as 
measured by: 

(1) message continuity, i.e., responding to other 
messages in context 

(2) concept elucidation and concept structure 
apprehension 

(3) divergent and convergent thinking processes 

(4) civility of message tone and toleration of 
diversity of opinion 

4. Learn to use technology. 

This goal will be measured in terms of: 

a. variety of software applications used 

b. the variety and frequency of methodological applications 
which result in knowledge production and analysis 

c. variety of data retrieval sources 

5. Learn to be an independent learner to acquire knowledge and 
solve problems. 

a. standardized tests of cumulative knowledge production and 
analysis 

b. comprehensive tests prepared by major departments 

In addition, student learning outcome goal number 6 states, in 
a generic way, the expected learning that will occur in the 
specific discipline or subject matter content of the course as 
described in the course syllabi. A pragmatic method of measurement 
related to this goal is defined below. 

6. Students enrolled in CNU Online courses will demonstrate, on 
average, development of knowledge and skills (as called for on 
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the course syllabi, specifically for the course regardless of 

the medium) comparable to that of students in the equivalent 

classroom courses. 

Measuring Learning Outcomes ; 

Essentially, the online program will require that the 
instructor build in assessment instruments of several types having 
differing objectives, as follows; 

1. to identify an appropriate learning format that supports 
a high level of student learning 

2. to identify what students already know at the start of a 
learning module 

3. to identify what students have learned at the end of a 
learning module 

4. to identify what can be certified as course learning at 
the end of a course 

These assessments will become part of the teaching process and 
will be integrated into the content of the course. Although how 
students learn and what they know or have learned can all affect 
course performance, except for item 4 these items will not directly 
impinge on the grading of individual students. 

As regards instruments and inquiry corresponding to item. 1, 
currently studies are being planned having to do with the 
interaction of learning styles and a variety of learning outcome 
measures. An initial study is planned for spring, 1995. This 
research will help to inform the CNU Online instruction in 
tailoring instruction to student characteristics and learning 
needs, as implied in item 1. In addition, the background data 
described earlier is closely related to this item since, as stated 
earlier, the information gained would be "essential to any attempt 
to account for background characteristics as a factor in student 
learning." 

Comparison of online courses and classroom courses can be 
difficult because of differences in teaching methodology. The 
differences are not total and they can be compared in areas where 
they are similar. Thus, pragmatic means to be described below will 
permit the application of items 2 and 3 above to this problem. 

The proposed method involves having instructors of online 
courses choose a classroom section of a classroom course to assess 
and compare the two types of courses where they overlap. 

The Online instructor and a classroom instructor teaching the 
same course will need to work together to implement this method, 
under the supervision of the department chairman. (If the 



instructor is teaching the course both ways, this requirement will 
be obviated.) The course content will be reduced to a set of 
concepts, terms, key words and phrases that the two instructors 
agree represent essential knowledge of the course content. A pre~ 
test on these concepts at the beginning of the course or module and 
a post-test at the end of the course or module will be 
administered. Questions will be randomly selected from the list of 
concepts in order to avoid selection bias. This approach will 
provide a pragmatic method of comparing courses taught in the two 
media on the basis of pre- and post-test comparisons in essential 
course content as defined above. 

A successful trial of the above approach will lead to its 
being implemented as a requirement for all online courses. 

In addition to course content learning, skill modules (e.g., 
technical writing) also can be assessed through this procedure. 

III. TEACHING EFFECTIVENESS LEVEL 

Peer Group Review of Effective Instruction 

Another level for assessment will involve the peer group 
procedures, another well-established institutional practice, in the 
review of effective online instruction. Although traditionally 
these procedures have much to do with personnel decisions and 
ittle or nothing to do with assessment, this plan proposes to 
involve the peer group members in making assessment-related 
ratings. These ratings will be used for program evaluation 
purposes (by means of the instrument to be described below) ; in 
addition the information supplied, and the associated ratings, can 
be considered by the peer groups in making their joint 
recommendations. The institutionalized procedure continues to 
provide checks on these recommendations, including the deans' and 
Provost's reviews. 

The peer group for each faculty person being evaluated will 
have available a copy of the message log for the courses taught by 
the candidate. (The log copy will not include restricted or 
student grade messages.) Specific citations in the message log 
will be used to support the conclusions drawn by the peer group. 
The instructor must document for the peer group specific examples 
in the course log supporting effectiveness in relation to the 
following items. This process itself can provide additional 
incentive to the instructor to derive the maximum educational 
benefits for students from this medium. 

The ratings will document the extent to which specific 
processes, designed to facilitate the achievement of course 
objectives, were in fact carried out by the instructor. Thus, this 
level of assessment deals directly with the processes or 
instrumentalities of achieving the course and program objectives. 
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A detailed rationale is included in the paper "Teaching Online, 
Computer Managed Conferencing," by Williams, Teschner and Miller. 
A copy of the rating scale questions appears below. 

Questions and Guidelines for Peer Group Review 

The peer group for each faculty person being evaluated will 
address at least the following factors. The peer group will have 
available a copy of the message log for the courses taught by the 
Candida ''.e. (The log copy will not include restricted or student 
grade messages.) Specific citations in the message log will be 
used to support the conclusions drawn by the peer group. The 
instructor must document for the peer group specific examples, in 
the course log, supporting numbers 1-5 below. Numbers 6-8 will be 
provided by CNU Online. (CNU Online will write software to 
generate data for numbers 6-8.) 



1. Did the instructor list the learning outcomes expected of each 
student at the completion of each reading assignment and at 
the end of the course? 

2. Did the instructor guide the student in a process of inquiry 
that required the student to manifest skills of comprehension, 
analysis, synthesis, application, and evaluation of the 
subject matter? 

3. Did the instructor assign the student to small groups of 
persons who collectively were required to create a group to 
negotiate differences in the production of a work product? 

4. Did the instructor integrate skill modules into the content of 
the course subject matter, as means of teaching skills (for 
example, technical writing, statistics, etc.) across the 
curriculum? 

5. Did the instructor require students to retrieve, analyze and 
interpret data using computer-managed resources? 

6. Did the instructor manage the learning environment in such a 
way that the substantial majority of messages in the log were 
peer-to-peer? 

7. Did students regularly upload from five to ten messages on 
average each week? 

8. Did the instructor logon the course conference to interact 
with students on the average of twice per day? 

9. Did the instructor's leadership result in at least fifty 
percent of the student messages meeting the following 
criteria: 
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- connectivity with the subject matter of other messages 

- dialogical quality of the message subject matter 

- conciseness of message word usage 

- elucidation of concepts and comprehension of conceptual 
frameworks 

- provocative and dialectical quality of message subject 
matter 

- divergent and convergent character of message content 

- civility of message tone and toleration of diversity of 
opinion 

In order to elicit appropriate quantitative data for each of 
the relevant questions, a rating scale form has been development 
(see Appendix B) . 

IV. STUDENT SATISFACTION 

Relevant Goal Statement 

Students enrolled in GNU Online courses will demonstrate, on 
average, satisfaction and motivation to persist and make progress 
in GNU Online courses comparable to that of students in the 
equivalent classroom courses. 

Assessment Method 



The first level is student satisfaction. The standard method 
for assessing student satisfaction in relation to GNU courses is 
the Instruction Evaluation Survey or lES. The procedures 
associated with this instrument are already well-integrated with 
institutional practice. The necessary steps and safeguards have 
now been taken resulting in implementation of an online version of 
this instrument during fall 1994. This mechanism is in place and 
can and should be used to maximum benefit for assessment purposes. 



The assessment methods using this mechanism are two: 

1. First, the average responses of students to relevant 
questions on the Instruction Evaluation Survey should be 
compared between online and classroom versions of the 
same course. Online and in class versions of that 
instrument are equivalent as far as the relevant 
questions are concerned. The departments concerned 
routinely will have access to the relevant results to 
review. 

2. Second, additional questions will be added to the online 
lES to gather student satisfaction data bearing on 
students' perception of their educational gains in the 
course and other concerns directly related to the goals 
of GNU Online. The list of additional questions are 
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found in Appendix A along with questions 1-22 of the 
original online lES. 

Aggregated results for each course will be prepared routinely 
in computer-generated report form each semester. Instructor 
identifications will be removed from these reports. These results 
will be made available to the Director of Assessment and Evaluation 
for reporting purposes and to the Provost, deans and relevant 
departments for program review purposes. 

Related to student satisfaction is student motivation to 
persist, or student retention, which is dealt with under the 
section dealing with Online Program/Institutional Efficiency. 

COLLATERAL METHODS 

The following activities will be carried out as time and 
resources peirmit in parallel with the other assessment methods 
described above and will be under the supeirvision of the Director 
of Assessment and Evaluation. 

1. Comparison of similar courses on-line and in the 

classroom, across the program, will be carried out using 
an innovative assessment technique. This technique 
examines overall grade performance of students in courses 
for which the prerequisites might have been taken either 
on-line or in a classroom. An equivalence of courses 
would be indicated by a similar distribution of grades 
regardless of the format of the prerequisite course. 
(This technique has been applied successfully to the 
study of transfer prerequisites versus CNU 

prerequisites . ) 

2. Another overall activity which may be referred to as 

assessment (in a broad sense) may be adopted depending on 
availability of staff and financial resources. This 

approach would involve having the same individual — under 
the supervision of the Director of Assessment and 
Evaluation — auditing the same class offered both ways. 
Detailed journals would be kept, with particular emphasis 
on the same or similar assignments. The instructor's (or 
instructors') cooperation would be sought. 

Uses of information from this source would be limited. The 
process would have no part in the external reporting to SCHEV. The 
Director of Assessment and Evaluation, in consultation as needed 
with CNU Online staff would note any unusual circumstances and make 
sure that atypical conditions would be properly understood as such. 
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3. A final potentially useful collateral activity will be 
referring occasional matters to an external consultant or 
consultants. The advice and recommendations received, 
while in no sense binding, may be well worth considering. 
The activity can also provide additional credibility in 
the form of external validity and scholarly integrity 
which need to characterize our efforts as we prepare to 
respond to external inquiries. 
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APPENDIX a: online INSTRUCTIONAL EVALUATION SURVEY (lES) WITH 
ORIGINAL QUESTIONS (1-22) AND ADDITIONAL QUESTIONS (23-39) 

Items number 1 through 14 use the following response choices: 

"Strongly Agree, Mildly Agree, Neutral, Mildly Disagree, and 

Strongly Disagree." 

1 . The course was consistent with the course description in the 
catalog. 

2. The instructor's messages showed evidence of careful thought 
and preparation. 

3. The instructor's grading policies for this course were clearly 
explained early in the term. 

4. The instructor's grading procedures were fair. 

5. Graded assignments and course contributions were responded to 
in a timely fashion. 

6. Grading covered knowledge, application, or reasoning that 
could be expected on the basis of course content. 

7. The instructor encouraged students to seek help when needed 
and was readily accessible when he or she did so. 

8. The instructor demonstrated command of the subject matter of 
the course. 

9. The instructor's messages to me and to the class made the 
course material clear and understandable. 

10. The instructor demonstrated interest and enthusiasm for the 
subject matter. 

11. I found the instructor to be intellectually motivating and 
stimulating. 

12. The instructor consistently devoted the time necessary to make 
this course a valuable learning experience for me. 

13. I found this professor to be an effective teacher in this 
course . 

14. The subject matter of this course is interesting. 

15. The subject matter of the course is a valuable part of roy 
education. 
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16. What is your present class standing? 

Response choices: freshman, sophomore, junior, senior 

17. Please mark one. 

Response choices: male, female 

18. Please mark one. 

Response choices: asian, black, native american, white, 

hispanic, other. 

19. What is your age? 

Response choices: 18 or less, 19-22, 23-30, over 30. 

20. What grade do you expect to receive in the course? 

Response choices: A,, B, C, D, F 

21. If you had the choice of taking the course online or in a 
classroom, which would it be? 

Response choices: online, in classroom 

22. What kind of requirement is this course? 

Response choices: distribution, major field, elective 

Additional c[uestions to measure student opinion related to 
assessment. All questions will be answered on a scale of 1 to 5 
(the same scale used for items 1-15) . 

23. The course developed my problem solving skills. 

24. The course enabled me to draw reasonable inferences from 
observations. 

25. The course developed my ability to integrate and synthesize 
information. 

26. The course developed my ability to use facts to support 
opinion. 

27. The course developed my ability to appreciate the historical 
development of the subject matter addressed in the course. 

28. The course developed my tolerance for other viewpoints. 

29. The course developed my ability to work productively with 
others. 
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The course developed my ability to resolve controversies 



31. The course helped me learn the vocabulary and concepts of the 
sub j ect . 

32. The course helped me learn the objectives and values of the 
sub j ect . 

33. The course developed my reading skills. 

34. The course developed my technical writing skills. 

35. The course developed my telecommunications skills. 

36. The course developed my computer software skills. 

37. It is more convenient for me to take this course onli le 
instead of in a classroom. 

38. Online courses are necessary for me to complete an 
undergraduate degree . 

39. Cost is a factor (scheduling, travel, etc.) in taking online 
courses . 



O 

ERIC 
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APPENDIX B: RATING SCALE FOR USE IN PEEL GROUP-BASED JURIED 
REVIEW OF RELEVANT ASPECTS OF INSTRUCTORS' MESSAGE LOGS 



Instructions ; For items 1-5, use the following 5-point scale 
in rating the evaluee's online message log with respect to the 
following dimensions: 

1 = little or no evidence 

2 = some evidence 

3 = good amount of evidence but not always consistent 

4 = strong evidence and consistency 

5 = exemplary evidence with respect to this 

characteristic 

1. The instructor listed the learning outcomes expected of each 
student at the completion of each reading assignment and at 
the end of the course. 

2. The instructor guided the student in a process of inquiry that 
required the student to manifest skills of comprehension, 
analysis, synthesis, application, and evaluation of the 
subject matter. 

3. The instructor assigned the student to small groups of persons 
who collectively were required to create a group to negotiate 
differences in the production of a work product. 

4. The instructor integrated skill modules into the content of 
the course subject matter as a means of teaching skills (for 
example, technical writing, statistics, etc.) across the 
curriculum. 

5. The instructor required students to retrieve, analyze and 
interpret data using computer-managed resources. 

For items 6-12, rate the evaluee on the approximate percent of 
student messages meeting the following criteria. Use the following 
scale: 



1 = 20 percent or less meet the criterion 

2 = 21 to 40 percent meet the criterion 

3 = 41 to 60 percent meet the criterion 

4 = 61 to 80 percent meet the criterion 

5 = 81 percent or more meet the criterion 

Items 6-12 are complete sentences beginning: "Student messages 

revealed" : 

6. connectivity with the subject matter of other messages. 

7. dialogical quality of the message subject matter. 
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8. conciseness of message word usage. 

9. elucidation of concepts and comprehension of conceptual 
frameworks. 

10. provocative and dialectical quality of message subject 
matter. 

11. divergent and convergent character of message content. 

12. civility of message tone and toleration of di\ersity of 
opinion. 



